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Program description^ 


Earobics® is interactive software that provides students in pre-K 
through third grade with individual, systematic instruction in early 
literacy skills as students interact with animated characters. Earo- 
bics® Foundations is a version for pre-Kindergarten, Kindergarten, 
and first graders. Earobics® Connections is for second and 
third graders and older struggling readers. The program builds 
children’s skills in phonemic awareness, auditory processing, and 


phonics, as well as the cognitive and language skills required for 
comprehension. Each level of instruction addresses recognizing 
and blending sounds, rhyming, and discriminating phonemes 
within words, adjusting to each student’s ability level. The soft- 
ware is supported by music, audiocassettes, and videotapes and 
includes picture/word cards, letter-sound decks, big books, little 
books, and leveled readers for reading independently or in groups. 


Research 


One study of Earobics® met the What Works Clearinghouse 
(WWC) evidence standards, and one met evidence standards with 
reservations. The studies included 104 students from grades K-3 
in Los Angeles and Chicago. Sixty-one students were English 


language learners.^ The WWC considers the extent of evidence 
for Earobics® to be small for alphabetics and fluency. No studies 
that met WWC evidence standards with or without reservations 
addressed comprehension or general reading achievement. 



Effectiveness 



The Earobics® program was found to have positive effects on alphabetics and no discernible effects on fluency. 







General reading 


Alphabetics 


Fluency 


Comprehension achievement 



Rating of effectiveness 


Positive effects 


No discernible effects na 


na 


Improvement index^ 


Average: +19 percentile 


Average: +4 percentile na 


na 




points 


points 






Range: 0 to +37 percentile 


Range: +3 to +6 percentile 





points points 

na = not applicable 

1. The descriptive information for this program was obtained from a pubiiciy avaiiabie source: the program’s web site fwww.earobics.com . downioaded 
Aprii 2007). The WWC requests deveiopers to review the program description sections for accuracy from their perspective. Further verification of the 
accuracy of the descriptive information for this program is beyond the scope of this review. 

2. The evidence presented in this report is based on avaiiabie research. Findings and conciusions may change as new research becomes avaiiabie. 

3. These numbers show the average and range of improvement indices for aii findings across the study. 
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Developer and contact 

Earobics® was developed in 1995 and is distributed by Houghton 
Mifflin Learning Technology. Address: P.O. Box 1363, Evanston, 

IL 60204-1363. Email: sales@earobics.com . Web: www.earobics. 
com . Telephone: (888) 328-8199. 

Scope of use 

According to the developers, Earobics® has been used nationally 
in more than 10,000 schools. The program has been used with 
at-risk students, general and special education students, and 
English language learners. 

Teaching 

The software is a supplemental program that can be used in 
conjunction with existing language arts programs. The Earobics® 
Teacher’s Guides help teachers plan students’ use of the soft- 
ware and supporting materials, using a teach, practice, and apply 



approach. As students work with the software, the program auto- 
matically adjusts based on each student’s performance. Reports 
on student performance can be printed or accessed online. 
Teachers may also customize the program for students, including 
selecting one of 10 languages for the directions. Teachers also 
have access to CD-ROMS with reproducible materials tied to 
specific lessons for students. Professional development for using 
Earobics® is available and focuses on instructional strategies to 
incorporate Earobics® into the curricula. 

Cost 

Currently, Earobics® Foundations and Earobics® Connections are 
available for either home use for $59 per user or a “clinic” version 
that accommodates up to 12 users for $299. Foundations is tar- 
geted for ages 4-7 and includes six interactive games with more 
than 300 levels of play. Connections is targeted for ages 7-10 and 
includes five interactive games with nearly 600 levels of play. 



Eighteen studies reviewed by the WWC investigated the effects 
of Earobics®. One study (Cognitive Concepts, 2003) was a ran- 
domized controlled trial that met WWC evidence standards. The 
other study (Valliath, 2002) was a quasi-experimental design that 
met WWC evidence standards with reservations. The remaining 
16 studies did not meet WWC evidence screens. 

Met evidence standards 

Cognitive Concepts (2003) conducted a randomized controlled 
trial of elementary school students in Los Angeles, California. 
Nineteen teachers identified students in Kindergarten through 
third grade with reading difficulties. Students were pretested, 
matched, and then randomly divided into two groups. In all, 39 



students used Earobics® in addition to Open Court, their regular 
reading curriculum, and 35 students in the comparison group 
used only Open Court. 

Met evidence standards with reservations 

Valliath (2002) is a quasi-experimental study of first-grade stu- 
dents from three elementary public schools in a high-achieving 
school district in Chicago, Illinois. Ten teachers each identified 
three children with the lowest reading ability within their respec- 
tive classrooms. Students were pretested, matched, and divided 
into two similar groups. In the analysis sample, 15 students used 
six exercises of the Earobics® software and 15 students in the 
comparison group used math software. 
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Extent of evidence 

The WWC categorizes the extent of evidence in each domain as 
small or moderate to large (see the What Works Clearinghouse 
Extent of Evidence Categorization Scheme) . The extent of 
evidence takes into account the number of studies and the total 
sample size across the studies that met WWC evidence standards 
with or without reservations."' 



The WWC considers the extent of evidence for Earobics® to 
be small for alphabetics and fluency. No studies that met WWC 
evidence standard with or without reservations addressed 
comprehension or general reading achievement. 



Findings 

The WWC review of interventions for beginning reading 
addresses student outcomes in four domains: alphabetics, read- 
ing fluency, comprehension, and general reading achievement.® 
The studies included in this report cover two domains: alphabet- 
ics and fluency. Within alphabetics, results for three constructs— 
phonological awareness, letter knowledge, and phonics— are 
reported. The findings below present the authors’ estimates 
and WWC-calculated estimates of the size and the statistical 
significance of the effects of Earobics® on students.® 

Alphabetics. Two studies reviewed findings in the alphabet- 
ics domain. Cognitive Concepts (2003) found and the WWC 
confirmed statistically significant positive effects on three pho- 
nological awareness measures (ORAL-J: Blending into Words, 
Segmenting into Sounds, and Rhyming Words subtests). The 
study authors did not find statistically significant effects of 
Earobics® on the letter knowledge measure (ORAL-J: Letter 
Naming subtest) or the phonics measure (the ORAL-J: Sound 
of Letters subtest).'' The average effect size across the five 



outcomes was large enough to be considered substantively 
important according to WWC criteria (that is, an effect size of at 
least 0.25). 

Valliath (2002) found that the overall intervention effect 
across the eight measures of beginning reading was not 
statistically significant.® The WWC analyzed four phonological 
awareness measures (Comprehensive Test of Phonological 
Processing (CTOPP): Blending Words, Blending Non-Words, 
Elision, and Sound Matching subtests) and two phonics mea- 
sures (Woodcock Reading Mastery Test: Word Identification 
and Word Attack subtests). The WWC found that the effect for 
one of the four phonological awareness tests (CTOPP: Sound 
Matching subtest) was positive and statistically significant. 
Effects for the other three phonological awareness and the two 
phonics subtests were not statistically significant. The average 
effect size across the six outcomes was large enough to be 
considered substantively important according to the WWC 
criteria (that is, an effect size of at least 0.25). 



4. The Extent of Evidence Categorization was deveioped to teii readers how much evidence was used to determine the intervention rating, focusing on the 
number and size of studies. Additionai factors associated with a reiated concept, externai vaiidity, such as the students’ demographics and the types of 
settings in which studies took place, are not taken into account for the categorization. 

5. For definitions of the domains, see the Beginning Reading Protocoi . 

6. The ievei of statisticai significance was reported by the study authors or, where necessary, caicuiated by the WWC to correct for ciustering within 
ciassrooms or schoois and for muitipie comparisons. For an explanation, see the WWC Tutorial on Mismatch . See the WWC Intervention Rating Scheme 
for the formulas the WWC used to calculate the statistical significance. In the case of Earobics®, corrections for multiple comparisons were needed. 

7. Data for some of the phonics outcomes were received through communication with the author. 

8. The WWC did not use all eight measures in its analysis. See Appendix A1 .2. 
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Effectiveness (continued) 



The WWC found Earobics^ 
to have positive 
effects on aiphabetics 
and no discernibie 
effects on fluency 



References 



Fluency. Cognitive Concepts (2003) did not find statistically 
significant effects of Earobics® and the effect was not large 
enough to be considered substantively important according to 
WWC criteria. 



Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as: positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 
design, the statistical significance of the findings, the size of 
the difference between participants in the intervention and the 
comparison conditions, and the consistency in findings across 
studies (see the WWC Intervention Rating Scheme) . 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and an 
average improvement index across studies (see Technical Details 
of WWC-Conducted Computations^ . The improvement index rep- 
resents the difference between the percentile rank of the average 
student in the intervention condition versus the percentile rank of 
the average student in the comparison condition. Unlike the rating 
of effectiveness, the improvement index is based entirely on the 
size of the effect, regardless of the statistical significance of the 
effect, the study design, or the analyses. The improvement index 
can take on values between -50 and +50, with positive numbers 
denoting results favorable to the intervention group. 



The average improvement index for aiphabetics is +19 
percentile points across the two studies, with a range of +0 to 
+37 percentile points across findings. The average improvement 
index for fluency is +4 percentile points in the one study, with a 
range of +3 to +6 percentile points across findings. 

Summary 

The WWC reviewed 18 studies on Earobics®. One study met 
WWC evidence standards, and one met evidence standards 
with reservations; the others did not meet WWC evidence 
screens. Based on the two studies, the WWC found positive 
effects on aiphabetics and no discernible effects on fluency. 
The evidence presented in this report may change as new 
research emerges. 



Met WWC standards 

Cognitive Concepts, Inc. (2003). Outcomes report: Los Angeles 
Unified School District, California. Retrieved from http://www. 
cogcon.com/research/proven/LAUSD.pdf 

Met WWC standards with reservations 

Valliath, S. (2002). An evaluation of a computer-based phonologi- 
cal awareness training program: Effects on phonological 
awareness, reading and spelling. Dissertation Abstracts 
International, 63(04), 1291A. (UMI No. 3050601) 



Did not meet WWC evidence screens 

Cognitive Concepts, Inc. (2000). Earobics Early Literacy Instruction: 
Chicago Public Schools pilot research report. Retrieved from 
http://www.cogcon.com/research/proven/cpsoutcomes.pdf® 
Cognitive Concepts, Inc. (2001). Outcomes report: Daviess 
County Public Schools, Kentucky. Retrieved from http://www. 
cogcon.com/research/proven/DaviessCounty.pdf® 

Cognitive Concepts, Inc. (2001). Outcomes report: Newport 
News Public Schools, Virginia. Retrieved from http://www. 
cogcon.com/research/proven/newportoutcomes.pdf® 



9. Does not use a strong causal design: this study did not use a comparison group. 
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For more information about specific studies and WWC caicuiations, piease see the WWC Earobics ® 
Technicai Appendices . 



10. The sample is not appropriate to this review: the parameters for this WWC review specified that students shouid be in grades K-3 during the time of the 
intervention; this study did not focus on the targeted grades. 

11. The outcome measures are not reievant to this review: the parameters for this WWC review specified student outcome measures but this study did not 
focus on students. 

12. High overaii attrition: the study, which used a randomized controiied triai design, reported an extreme overaii attrition rate. 

13. The sample is not appropriate to this review: this study did not disaggregate data for students in other grades K-3, the focus of this WWC review. 
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Appendix 



Appendix A1.1 Study characteristics: Cognitive Concepts, 2003 (randomized controiied triai) 



Characteristic 


Description 


Study citation 


Cognitive Concepts, inc. (2003). Outcomes report: Los Angeles Unified School District, California. Retrieved from http://www.cogcon.com/research/proven/LAUSD.pdf 


Participants 


Nineteen teachers identified sfudenfs in Kindergarten through third grade with reading difficuities. More than 80% of students were Engiish ianguage iearners. The study 
author administered pretests (ORAL-J and Test of Memory and Learning (TOMAL)) to students to divide them into two simiiar groups.'' The groups were then randomiy 
assigned to be either the intervention or comparison group. Each group originaiiy had 43 students, but there was some attrition due to poor attendance.^ in the anaiysis 
sampie, 39 students were in the intervention group and 35 students were in the comparison group. 


Setting 


The study took piace in one eiementary schooi in Los Angeies, Caiifornia. 


Intervention 


Students in the intervention group were given directions on how to use Earobic^ software. They received instruction with Earobic^ for 30 minutes a day, five days a week 
over three months, in addition, the intervention group received the reguiar whoie ciass reading instruction with the Open Court Reading cumculum. 


Comparison 


Students in the comparison ciasses received the reguiar whoie ciass reading instruction with the Open Court Reading curricuium during the ianguage arts period. 


Primary outcomes 
and measurement 


Eor both pre- and posttests, the authors administered six subtests of the ORAL-J Eariy Literacy Achievement test: Biending into Words, Segmenting into Sounds, Rhyming 
Words, Letter Naming, and Sound of Letters subtests as weii as three administrations of the Words per Minute subtest.^ The Test of Memory and Learning (TOMAL) was aiso 
used in the study, but was not inciuded in this review because it was outside the scope of the Beginning Reading review (see Appendices A2.1-2.2 for more detaiied descrip- 
tions of outcome measures). 


Teacher training 


No information on teaching training is provided. The Earobic^ group worked with minimai teacher supervision in a computer iab. 



1 . Equivalence of the two groups at pretest was confirmed through data sent by the author, M. Poblanz. 

2. Some information about attrition was provided through personal communication with the author. 

3. Some of the test data was not in the published report and was provided directly by the author. 
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Appendix A1.2 Study characteristics: Vaiiiath, 2002 (quasi-experimentai design) 



Characteristic 


Description 


Study citation 


Vaiiiath, S. (2002). An evaluation of a computer-based phonological awareness training program: Effects on phonological awareness, reading and spelling. Dissertation 
Abstracts Intemationai, 63(04), 1291A. (UMI No. 3050601) 


Participants 


Ten teachers each identified three children with the lowest reading ability within their first-grade classrooms. The study author administered a pretest (the Woodcock Reading 
Mastery Test: Word Identification subtest) to students to divide them into two similar groups.'' All children came from English-speaking monolingual homes; none received any 
special education or speech and language services. The sample consisted of 16 boys and 14 girls, ranging in age from 6.5 to 7.5 years. In the analysis sample, 15 students 
were in the intervention group and 15 students were in the comparison group. 


Setting 


The study took place in three public elementary schools from a high-achieving school district in a northwest suburb of Chicago, Illinois. 


Intervention 


Students in the intervention group spent 20 minutes each day playing one of the six Earobic^ games. Earobic^ delivered phonological awareness training in the auditory 
mode and provided minimal sound-to-print training. The children played individually and were provided headsets. They started at the lowest skill level for each game and 
progressed at their own pace. The games were rotated systematically on a daily basis during the ten-week training program. The average number of days attended by the 
students in intervention group was 46.47 of a possible 50 days. 


Comparison 


Students in the comparison classes received comparable amounts of daily exposure (approximately 20 minutes) to math training software, Knowiedge Adventure’s Jump Start 
Math for First Graders. The software has no linguistic training component and consists of eight math games appropriate for children in the first grade. The average number of 
days attended by the students in comparison group was 45.8 of a possible 50 days. 


Primary outcomes 
and measurement 


Eor both pre- and posttests, the authors administered the Comprehensive Test of Phonological Processing (CTOPP): Blending Words, Blending Non-words, Elision, and Sound 
Matching subtests and the Woodcock Reading Mastery Test: Word Identification and Word Attack subtests. The CTOPP Memory for Digits subtest and the Spelling subtest 
of the Wide Range Achievement Test were also used in the study but have not been included because they were outside the scope of the Beginning Reading review (see 
Appendices A2.1-2.2 for more detailed descriptions of outcome measures). 


Teacher training 


The experimenter trained the computer lab technicians in each of the three schools on how to use the software. Detailed instructions, attendance sheets, and appropriate 
rotations of the Earobicd^ games were discussed. No other information on teaching training is provided. 



1 . The pretest also confirmed that students’ performance was low-average. 
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Appendix A2.1 Outcome measures in the aiphabetics domain by construct 



Outcome measure 


Description 


Phonological awareness 

Comprehensive Test of Phonological 
Processing (CTOPP); Elision subtest 


This subtest measures the ability of a child to manipulate the components of a word. The examinee is prompted to say a compound word (for example, "cowboy”) and 
then to say the word without the first part "cow”, or to say the word without a specific sound (such as "f” in "farm” - “arm”) (as cited in Valliath, 2002). 


CTOPP; Blending Words subtest 


This subtest measures the ability of a child to combine separately spoken sounds and put them together to form a real word (as cited in Valliath, 2002). 


CTOPP; Sound Matching subtest 


This subtest measures the ability of a child to choose the word that contains a target sound. Words are presented orally and the subject is shown a card containing 
pictures of the four words. The child must indicate which word contains the sound. The target sound is tested both in the initial and final position in the word (as cited 
in Valliath, 2002). 


ORAL-J; Early Literacy Achievement 
Blending into Words subtest 


This task requires children to combine or blend the separate sounds of a word to say the word. For example, child is given sounds like /k/ /a/ /t/ and has to say "cat” 
(as cited in author communication^). 


ORAL-J; Early Literacy Achievement 
Segmenting into Sounds subtest 


This task requires students to segment words into sounds. The child is given a word and has to give individual sounds (as cited in author's communication^). 


ORAL-J; Early Literacy Achievement; 
Rhyming Words subtest 


This task requires students to generate words that rhyme. The child is given a word and has to supply a word that rhymes (as cited in author's communication^) 


CTOPP; Blending Non-Words subtest 


This subtest measures the ability of a child to combine sounds that are presented orally and put the separate sounds together to form a nonsense word (as cited in 
Valliath, 2002). 


Letter knowledge 

ORAL-J; Early Literacy Achievement; 
Letter Naming subtest 


Children get a card with 100 letters and are to name each one. The subtest score is determined by how many letters they name in a minute (as cited in author's 
communication^). 


Phonics 

ORAL-J; Early Literacy Achievement; 
Sound of Letters subtest 


Children name the sound of letters on a card with 59 letters. The subtest score is determined by how many sounds they name in a minute (as cited in author's 
communication^ 


Woodcock Reading Mastery Test 
(WRMT); Word Identification subtest 


This subtest of the standardized WRMT measures basic word reading skills and requires the child to read aloud isolated words that range in frequency and difficulty 
(as cited in Valliath, 2002). 


WRMT; Word Attack subtest 


The Word Attack subtest of the WRMT measures the child's ability to apply phonic and structural analysis skills to pronounce unfamiliar words. Subjects cannot read 
the pseudowords by sight and must rely on phonological processes to decode them (as cited in Valliath, 2002). 



1 . The information was received from M. Pobianz, author of Cognitive Concepts (2003). 
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Appendix A2.2 Outcome measure in the fluency domain by construct 



Outcome measure 


Description 


ORAL-J; Early Literacy 
Achievement Words per Minute 


As students read a story, the teacher times their word recognition for one minute. There are three different stories on each ORAL-J test from which three different 
WPM scores are derived (as cited in author’s communication^). 



1. The information was received from M. Pobianz, author of Cognitive Concepts (2003). 
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Appendix A3.1 



Summary of study findings inciuded in the rating for the aiphahetics domain hy construct^ 



Authors’ findings from the study 



Mean outcome 

(standard deviation^) WWC calculations 









Earobic^ 




Mean difference'* 




Statistical 






Study 


Sample size 


Comparison 


{Earobic^ - 




significance^ 


Improvement 


Outcome measure 


sample 


(students) 


groups 


group 


comparison) 


Effect size^ 


(at a = 0.05) 


index^ 



Construct: Phonological awareness 

ORAL-J: Blending into Grades K-3 

Words subtest® 


74 


Cognitive Concepts, 2003 (randomized controlled trial)^ 

17.31 14.86 2.45 

(3.54) (4.10) 


0.64 


Statistically 

significant 


-h24 


ORAL-J: Segmenting 
into Sounds® 


Grades K-3 


74 


45.31 

(14.31) 


35.80 

(15.82) 


9.51 


0.63 


Statistically 

significant 


-h23 


ORAL-J: Rhyming Words® 


Grades K-3 


74 


7.16 

(5.31) 


4.26 

(4.36) 


2.90 


0.59 


Statistically 

significant 


-h22 








Valliath, 2002 (quasi-experimental design)^ 










OTOPP: Eiision 


First grade 


30 


104.00 


97.67 


6.33 


0.63 


ns 


-h23 








(11.98) 


(7.04) 










OTOPP: Biending Words 


First grade 


30 


105.66 


103.33 


2.33 


0.29 


ns 


-Hi 2 








(4.88) 


(9.76) 










OTOPP: Sound Matching 


First grade 


30 


103.63 


95.00 


8.63 


1.13 


Statistically 


-h37 








(4.58) 


(9.45) 






significant 




OTOPP: Blending Non-words 


First grade 


30 


111.00 


105.33 


5.67 


0.57 


ns 


-h22 








(9.02) 


(10.26) 










Construct Letter knowledge 
























Cognitive Concepts, 2003 (randomized controlled trial)^ 








ORAL-J: Letter Naming® 


Grades K-3 


74 


57.49 


57.26 


0.23 


0.01 


ns 


0 








(18.78) 


(20.63) 










Construct Phonics 
























Cognitive Concepts, 2003 (randomized controlled trial)^ 








ORAL-J: Sound of Letters® 


Grades K-3 


74 


27.80 


26.17 


1.63 


0.22 


ns 


-h9 








(6.89) 


(7.72) 











(continued) 
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Appendix A3.1 Summary of study findings inciuded in the rating for the aiphahetics domain hy construct (continued) 









Authors’ findings from the study 














Mean outcome 
(standard deviation^) 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(students) 


Earobic^ Comparison 

groups group 


Mean difference'* 
{Earobic^ - 
comparison) 


Statistical 
significance^ 
Effect size® (at a = 0.05) 


Improvement 

index^ 







Valliath, 2002 (quasi-experimental design)® 








WRMT: Word Identification First grade 


30 


104.07 

(5.16) 


100.87 3.2 

(3.76) 


0.69 


ns 


+25 


WRMT: Word Attack First grade 


30 


103.33 

(6.18) 


101.33 2.0 

(6.90) 


0.30 


ns 


+12 


Average^® for aiphahetics domain (Cognitive Concepts, 2003) 






0.42 


ns 


+16 


Average^® for aiphahetics domain (Valliath, 2002) 








0.60 


ns 


+23 


Domain average^® for aiphahetics 








0.51 


na 


+19 



ns = not statistically significant 
na = not applicable 

1 . This appendix reports findings considered for the effectiveness rating and the average improvement index. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 

3. The farab/cs® group mean equals the comparison group mean plus the mean difference. The computation of the mean difference took into account the pretest difference between the study groups. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition versus the percentile rank of the average student in the comparison condition. The improvement index 
can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

8. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the 
clustering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formula the WWC used to calculate statistical significance. In the case of Cognitive Concepts (2003) and Valliath 
(2002), corrections for multiple comparisons were needed, so the significance levels may differ from those reported in the original studies. 

9. Means and standard deviations were received through communication with the author. 

1 0. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect size. 
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Appendix A3.2 Summary of study findings inciuded in the rating for the fluency domain^ 









Authors’ findings from the study^ 














Mean outcome 
(standard deviation^) 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(students) 


Earobic^ Comparison 

group'* group 


Mean difference^ 
{Earobic^ - 
comparison) 


Statistical 
significance^ 
Effect size® (at a = 0.05) 


Improvement 

index® 



Cognitive Concepts, 2003 (randomized control trial)^ 



ORAL-J: Words per Minute 1 


Grades K-3 


74 


39.21 

(22.95) 


35.49 

(26.32) 


3.72 


0.15 


ns 


+6 


ORAL-J: Words per Minute 2 


Grades K-3 


74 


34.11 

(25.91) 


31.63 

(33.64) 


2.48 


0.08 


ns 


+3 


ORAL-J: Words per Minute 3 


Grades K-3 


74 


36.70 

(27.35) 


33.86 

(32.02) 


2.84 


0.10 


ns 


+4 



Domain average^" for fluency (Cognitive Concepts, 2003) 0.11 ns +4 



ns = not statistically significant 

1 . This appendix reports findings considered for the effectiveness rating and the improvement index. 

2. Means and standard deviations were received through communication with the author. 

3. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 

4. The Earobic^ group mean equals the comparison group mean plus the mean difference. The computation of the mean difference took into account the pretest difference between the study groups. 

5. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

6. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

7. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

8. The improvement index represents the difference between the percentile rank of the average student in the intervention condition versus the percentile rank of the average student in the comparison condition. The improvement index 
can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

9. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clus- 
tering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Cognitive Concepts (2003), a correction 
for multiple comparisons was needed, so the significance levels may differ from those reported in the original study. 

10. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect size. For a single 
study included in the fluency domain, the study average is equal to domain average. 



WWC Intervention Report Earobics® 



August 13, 2007 



12 





Appendix A4.1 



Earobic^ rating for the aiphabetics domain 



The WWC rates an intervention’s effects in a given outcome domain as positive, potentiaily positive, mixed, no discernibie effects, potentialiy negative, or negative.'' 

For the outcome domain of aiphabetics, the WWC rated Earobics® as having positive effects. The remaining ratings (potentiaiiy positive effects, mixed effects, no 
discernible effects, potentially negative effects, and negative effects) were not considered because Earobics® was assigned the highest applicable rating. 



Rating received 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Met. Two studies showed statistically significant positive effects, and one study had a strong design. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. No study showed statistically significant or substantively important negative effects. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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Appendix A4.2 Earobic^ rating for the fluency domain 



The WWC rates an intervention’s effects in a given outcome domain as positive, potentiaily positive, mixed, no discernibie effects, potentialiy negative, or negative.'' 

For the outcome domain of fluency, the WWC rated Earobics® as having no discernible effects. It did not meet the criteria for other ratings (positive effects, 
potentially positive effects, mixed effects, potentiaiiy negative effects, and negative effects) because the one study that met WWC standards did not show statisticaily 
significant or substantively important effects. 



Rating received 

No discernible effects: No affirmative evidence of effects. 

• Criterion 1: None of the studies shows a statisticaiiy significant or substantively important effect, either pos/f/ve or negative. 

Met. No study showed a statisticaiiy significant or substantiveiy important effect, either positive or negative. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statisticaiiy significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No studies show statisticaiiy significant positive effects. 

AND 

• Criterion 2: No studies showing statisticaiiy significant or substantively important negative effects. 

Met. No study showed statistically significant or substantiveiy important negative effects. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At ieast one study showing a statisticaiiy significant or substantively important positive effect. 

Not met. No study showed a statisticaiiy significant or substantively important positive effect. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Not met. No study showed a statistically significant or substantively important negative effect, but one study showed indeterminate effects. 

Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect, and at least one study showing a statistically significant 
or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important positive effect. 

Not met. No study showed a statistically significant or substantively important effect, either positive or negative. 

OR 

• Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing a 
statistically significant or substantively important effect. 

Not met. No study showed a statistically significant or substantively important effect, while one study showed indeterminate effects. 

(continued) 
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Appendix A4.2 Earobic^ rating for the fluency domain (continued) 



Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1: At ieast one study showing a statisticaiiy significant or substantively important negative effect. 

Not met. No study showed a statistically significant or substantively important negative effect. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important positive effect, or more studies showing statistically significant or substantively 
important negative effects than showing statistically significant or substantively important positive effects. 

Met. No study showed a statistically significant or substantively important positive effect. 

Negative effects: Strong evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant negative effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No study showed a statistically significant or substantively important negative effect. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important positive effects. 

Met. No study showed statistically significant or substantively important positive effects. 



1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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Appendix A5 Extent of evidence by domain 



Outcome domain 


Number of studies 


Schoois 


Sampie size 


Students 


Extent of evidence^ 


Alphabetics 


2 


4 




104 


Small 


Fluency 


1 


1 




74 


Small 


Comprehension 


0 


0 




0 


na 


General reading achievement 


0 


0 




0 


na 



na = not applicable/not studied 

1 . A rating of “moderate to large” requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. 
Otherwise, the rating is “small.” 
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